Method for operating a head-mounted display apparatus in a motor vehicle, control device, and head-mounted display apparatus

ABSTRACT

A control device in a motor vehicle receives a signal from a sensor device which includes at least one image of the current interior situation. The control device generates a superimposition signal containing the at least one image of the current interior situation and transmits the superimposition signal to a display element of a display apparatus. The display apparatus superimposes the at least one image of the current interior situation onto predefined output content output to a user.

The invention relates to a method for operating a head-mounted displaydevice in a motor vehicle.

With the advance of the development of autonomously driving motorvehicles, head-mounted output devices have become increasingly importantfor entertainment during a journey and also for the currently growingdevelopment of driving services. The use of immersive on-boardentertainment in particular for example in ride-hailing vehicles isfaced with the challenge that the user can no longer be aware of thesituation in the vehicle when wearing, for example, VR glasses. Whilethis is not an issue in normal passenger entertainment, this may verywell become a problem for example in a ride-hailing vehicle, where theuser possibly wishes to be fully aware of the situation so as to feelmore at ease and safe.

While immersing into a virtual reality (“VR”) in such a vehicle, forexample, the user for example no longer has awareness of what the driverdoes or what other passengers, who are typically strangers, do. It isnot possible to see if, for example, the unfamiliar driver wishes tospeak to the passenger and if the driver is looking at the passenger. Ifthe driver addresses the passenger, this can only be perceived if thedisplay device leaves the ears uncovered and the display device has noearphones for a sound experience of an entertainment program. If thepassenger for example is also unable to acoustically perceive theunfamiliar driver, the only option the driver has is for example to tapthe shoulder of the user of the display device or to touch their arm,which could not only be very uncomfortable for the user of the displaydevice, but it might startle the user of the display device.

DE 10 2014 010 309 A1 describes an apparatus having a display unit thatis to be worn by the user on the head, which display unit is embodied tocompletely shield the user's eyes against light with respect to asurrounding area and to stereoscopically display in front of the eyes avirtual object disposed in a virtual space.

DE 10 2009 049 073 A1 discloses a method for presenting virtualinformation in a view of a real surrounding area.

DE 10 2014 213 285 A1 describes a method for displaying contents onsmartglasses.

One option for circumventing the problems addressed above is to omitearphones, but this means that the user has no sound for the virtual orpartially virtual entertainment program, resulting, for example, in avirtual reality or an augmented reality not being very immersive.

One object on which the invention is based is to increase a feeling ofsafety and the level of comfort in the motor vehicle while using anentertainment offering.

The stated object is achieved by the method according to the inventionand the apparatuses according to the invention. Advantageousdevelopments are evident from the dependent claims.

The invention is based on the idea of, in addition to the actual outputcontent of an entertainment offering, that is to say in addition tooutputting the main content, superposing an image (and/or a capturedimage series) of the interior of the motor vehicle captured by one ormore sensors when using a head-mounted display device in the motorvehicle. The image or the image series is in particular a current imageor a current image series. The disadvantages discussed above aresignificantly reduced or even eliminated. A user of a head-mounteddisplay device feels safe because, despite immersing themselves into forexample the virtual reality, they have awareness of the interior of themotor vehicle. Uncomfortable situations in which said user is touched bya stranger without warning can be effectively prevented.

The method according to the invention for operating a head-mounteddisplay device, for example augmented reality glasses, mixed realityglasses, or virtual reality glasses, while the head-mounted displaydevice is located in an interior of a motor vehicle, is accordingly amethod for operating the head-mounted display device in the interior ofthe motor vehicle. The display device is configured to output aspecified output content describing an entertainment offering, forexample a computer game or a movie. The output content can be specifiedfor example by the user selecting the output content and thus theentertainment offering or by the output content being specified by adefault setting.

The method is performed by a controller, wherein a controller isunderstood to mean a device or a device component for receiving andevaluating signals and for generating control signals. Accordingly, themethod can be referred to as a computer-implemented method for operatingthe head-mounted display device. The controller can be designed forexample as a control chip or a control device. The controller canpreferably be a controller of the motor vehicle or of the head-mounteddisplay device.

The controller receives a sensor signal from a sensor device, whereinthe received sensor signal describes at least one image of the currentinterior situation of the motor vehicle. A sensor device is understoodto mean a device or a device group for capturing properties of theenvironment, in particular for capturing a property of the interior,which is why the sensor device can preferably comprise for example atleast one interior camera of the motor vehicle, and/or for example atleast one infrared sensor. The at least one image can be, for example, aphotograph, a thermal image, or an image series, for example a film ofwhat is happening in the interior.

The controller generates a superposition signal, which describes thesuperposition of the at least one image of the current interiorsituation during the output of the specified output content andtransmits the generated superposition signal to a display element of thedisplay device and thereby causes the display device to superpose the atleast one image of the current interior situation during the output ofthe specified output content. A display element is a structural part orstructural part group or device component for displaying image contents,for example a screen. In other words, the controller causes the displaydevice to output the at least one image of the interior of the motorvehicle in addition to the output content, for example in addition to avirtual reality, wherein the at least one image of the interior isoutput preferably simultaneously with the output content, that is to saythe at least one image of the interior is superposed into only a partialregion of the visual field of a user of the display device.

This offers the advantages described above.

Optionally, the controller can additionally make available the specifiedoutput content. This can be desired for example if the controller is acontroller of the motor vehicle and the output content is made availableby the motor vehicle for example in the form of a type of entertainmentservice. However, the controller can optionally make available thespecified output content even if the controller is a controller of thedisplay device.

According to an embodiment of the method according to the invention, thegenerated superposition signal can describe the superposition of the atleast one image of the current interior situation into only a partialregion of a visual field of the user of the display device, preferablyin a primary or central visual field. The user can consequentlyimmediately capture the interior situation.

The received sensor signal can optionally describe at least one cameraimage of an interior camera of the sensor device. The user can evenrecognize for example micro gestures and facial expressions of otheroccupants on this superposed image.

The controller can preferably receive the sensor signal from a sensordevice of the motor vehicle. The interior can thereby be recorded fromdifferent perspectives, and the method is also compatible withhead-mounted display devices that do not possess a corresponding sensorsystem.

The specified output content can describe for example a virtual reality,in particular a 360-degree virtual reality. This ensures that the userof the display device has a particularly immersive experience during thejourney.

According to a preferred embodiment of the method according to theinvention, the controller can generate the superposition signal, ortransmit it, only if the controller ascertains that an occupant isspeaking, preferably an occupant other than the user of the displaydevice. It is possible for this purpose, for example, for directionalmicrophones to be installed in the motor vehicle and to be coupled tothe controller.

The superposition signal can preferably additionally be generated ortransmitted in dependence on the volume of the ascertained speakingvoice. Consequently, the user is made aware of the interior exactly whenthere is a high likelihood that the other person would like to speak tothe user of the display device. However, the user possibly continues tohear the sound of the output content but is able to recognize from thesuperposed image that someone is addressing them right now, and they candecide whether they would like to briefly remove the glasses. Sincewhatever the other occupant is saying under certain circumstances doesnot penetrate to the user of the display device in this embodiment ofthe method, any audio experience is not disturbed.

According to a further preferred embodiment of the method according tothe invention, the controller can generate the superposition signal, ortransmit it to the display element of the display device, only if thecontroller ascertains that the motor vehicle is approaching adestination of the motor vehicle, for example when a distance from thedestination falls under a specified threshold value or when specifiedgeographic coordinates are reached. The user is thus “pulled” out oftheir virtual reality only if a likelihood exists that the driver wouldsoon like to speak to them, for example because of payment for thejourney, or that it is likely that the user needs to gather theirbelongings because they are about to arrive at the destination.

In one variant, the controller can generate the superposition signal, ortransmit it to the display element of the display device, only if thecontroller receives an activation signal from a motor vehicle, forexample a signal describing that the hazard warning lights have beenswitched on. Alternatively, the at least one image of the interiorsituation can be switched off if the controller receives a deactivationsignal from the motor vehicle.

Preferably, the controller can additionally receive a sensor signal fromthe sensor device that describes at least one image of an outside areaaround the motor vehicle, which can preferably be received from anexternal camera. Either the superposition signal generated by thecontroller can additionally describe the at least one image of theoutside area, or the controller can generate a further superpositionsignal describing the at least one image of the outside area. The imageof the outside area can then for example be superposed at a differentlocation than the image of the interior. The user of the display devicetherefore additionally has awareness of the surrounding area of themotor vehicle. The sensor device can for this purpose have a cameraarranged on an outer skin or on a roof of the motor vehicle.

According to a further embodiment, the controller can generate a mapimage signal that can describe an image of a navigational map with, forexample, the current position of the motor vehicle. This image can bedisplayed by the display device in addition to the image of the interiorsituation, and the controller can for this purpose transmit thegenerated map superposition signal to the display element of the displaydevice.

Alternatively, the map image signal can be output for example onlywithin a specified time period, for example only at the start and/oronly at the end of the journey, or only within a specified time periodafter the start of the journey and before the end of the journey. Theinterior image can then be displayed only if the image of thenavigational map is switched off.

The object stated above is achieved by a controller that is configuredto implement one or more embodiments of the method according to theinvention. The controller can preferably have a processor device and/ora data memory. A processor device is here understood to mean a devicecomponent for electronic data processing, which has at least onemicrocontroller and/or one microprocessor. Preferably, a program codefor carrying out the method according to the invention can be stored onthe optional data memory. The program code can then be designed tocause, upon execution by the processor device, the controller to carryout one of the above-described embodiments of the method according tothe invention. The controller can then preferably be in the form of acontrol chip, control device, or application program (“app”).

The problem stated above is achieved by a motor vehicle having anembodiment of the controller according to the invention. The motorvehicle can preferably be a car-like vehicle, in particular a passengercar or truck, or a passenger bus or motorcycle.

The object stated above is achieved by a head-mounted display devicehaving an embodiment of the controller according to the invention. Thehead-mounted display device, also known as a “head-mounted display”, isa visual output device that is worn on the head. It is designed andconfigured to either present images on a screen close to the eyes or toproject the images directly onto the retina. The head-mounted displaydevice can preferably be virtual reality glasses, augmented realityglasses, or mixed reality glasses.

The invention also encompasses developments of the controller accordingto the invention, of the motor vehicle according to the invention, andof the display device according to the invention, which have features ashave already been described in connection with the developments of themethod according to the invention. For this reason, the correspondingdevelopments of the control device according to the invention, of themotor vehicle according to the invention, and of the display deviceaccording to the invention will not be described again.

The invention also encompasses the combinations of the features of thedescribed embodiments.

Exemplary embodiments of the invention will be described below. In thisregard, the single FIGURE (“FIGURE”) shows a schematic illustration of afirst exemplary embodiment of the method according to the invention andof the apparatuses according to the invention.

The exemplary embodiments explained below are preferred embodiments ofthe invention. In the exemplary embodiments, the described components ofthe embodiments each represent individual features of the invention thatshould be considered independently of one another and that develop, alsoindependently from one another, the invention in each case. Therefore,the disclosure is also intended to encompass other combinations of thefeatures of the embodiments than those illustrated. The describedembodiments can further also be supplemented by further features of theinvention already described.

In the figures, identical reference signs in each case refer to elementshaving the same functions.

The FIGURE illustrates the principle of the method according to theinvention and of the apparatuses according to the invention withreference to a first exemplary embodiment. In this regard, the FIGUREshows a motor vehicle 10, for example a passenger car of a ride-hailingdriving service. In the example of the FIGURE, the controller 12 can be,for example, a control device or a control chip of the motor vehicle 10but can alternatively be installed in the head-mounted display device14. The head-mounted display device 14 can have for example a screen asthe display element (not shown in the FIGURE).

Communication with the display device 14 takes place via a datacommunication link 16. For example, if the controller 12 is installed inthe display device 14, the data communication link 16 can be awire-based data communication link, for example a cable or a databus. Ifthe controller 12 is a device component of the motor vehicle 10, thedata communication link 16 for communicating between the controller 12and the display device 14 can likewise be a wire-based datacommunication link 16, for example a cable, but is preferably a wirelessdata communication link 16, for example a WLAN, mobile radio, Internet,or Bluetooth connection. The controller 12 of the FIGURE here shows anoptional processor device 20 and also an optional data memory 22.

If the controller 12 is a component of the motor vehicle 10, the datacommunication link 16 to the sensor device 18, which is illustrated inthe FIGURE as an interior camera, can be implemented for example via adatabus of the motor vehicle 10, for example a CAN bus.

In the example of the FIGURE, a user of the exemplary ride-hailingdriving service of the motor vehicle 10 can have booked, for example, ajourney of ten kilometers or a journey with a driving time of half anhour. The entertainment by means of the head-mounted display device 14can for example constitute an additional service of this drivingservice. The user can either have brought along their own display device14, or the display device 14 can be provided by the driving service. Forthe journey, for example, an entertainment offering that can describe atrip in a virtual reality can be provided during the journey, inparticular a 360-degree virtual reality. The output content can be, forexample, a Jurassic Park-based virtual reality, that is to say a form ofentertainment that describes a virtual trip through a prehistoriclandscape.

The output of the specified output content can begin for example at thestart of the journey. In method step S1, the controller 12 receives atleast one sensor signal that ideally describes the interior from aplurality of interior cameras of the motor vehicle 10. Although theFIGURE shows by way of example only one camera of the sensor device 18,it is ideally possible for a plurality of cameras to be arranged suchthat the interior is filmed, for example, from different perspectives.The images or image series of the individual cameras can then either bedescribed in one sensor signal or in a plurality of sensor signals.

The superposition signal generated in method step S2 can thus describe acamera image or the camera images of a plurality of interior cameras.After the generated superposition signal has been transmitted (S3) tothe display element of the display device 14, the latter is caused tosuperpose (S4) the corresponding image or the corresponding imagespreferably into a peripheral visual field of the user. If the controller12 is a controller 12 of the motor vehicle 10, the generatedsuperposition signal is thus also first transmitted to the displaydevice 14 (S3).

Owing to the superposed image (or owing to the superposed images orimage series), the user is able to see for example other, unfamiliarpassengers and the unfamiliar driver. If for example the driver wishesto say something to the user, for example if the driver has a question,the user of the display device 14 sees that the driver is turning towardthem. Even if the user of the display device 14 additionally receivesthrough the display device 14 audio content from the virtual reality,for example via earphones, said user can recognize that the driverwishes to speak with them. So if the driver initially receives no replyfrom the user and decides to shake the shoulder of the user of thedisplay device 14, this will not be unexpected for the user and the userof the display device 14 will not be startled. In addition, the user ofthe display device 14 is able to react much earlier anyway and canprevent the shaking of the shoulder, for example by taking off thedisplay device 14 in order to speak to the driver.

In order to avoid constantly drawing the attention of the user of thedisplay device 14 to the superposed image, but to do so only in specificsituations, the controller 12 may generate the superposition signal (S2)or transmit it (S3) only if the controller 12 ascertains (S5), forexample by means of a microphone (not shown in the FIGURE), that forexample the driver has just started to speak. It is optionally possiblehere for a threshold value for a volume to be additionally stored, tothe effect that the image is for example not yet superposed if thedriver merely mumbles quietly to themselves.

Alternatively or additionally, the superposition signal can be generated(S2) or the generated superposition signal can be transmitted (S3) onlyif the controller 12 ascertains (S6), for example on the basis of GPSdata or of navigational data ascertained using a navigation device ofthe motor vehicle 10, that the motor vehicle 10 is moving for example ata distance of only one kilometer from the destination. If the interiorsignal is superposed only at that time, the user of the display device14 will know that they are nearly there and for this reason is betterable to understand that any agitation they felt in the interior of themotor vehicle 10 is due to the fact that other, unfamiliar passengersare beginning to gather their belongings.

Overall, the examples show how the invention can be used to provide ahead-mounted display device 14, preferably VR glasses.

In a further example it is possible, for example when using VR glassesin the motor vehicle 10, to superpose (S4) into the exemplary VRenvironment, in addition to the main content, a (for example generallysmall) camera image that can communicate the situation in the vehicleinterior and possibly also outside the motor vehicle 10 to the VR user,with the result that the latter will always have an awareness of thecurrent situation in and around the motor vehicle 10.

1-10. (canceled)
 11. A method for operating a head-mounted displaydevice while the head-mounted display device is located in an interiorof a motor vehicle and is configured to output specified output contentof an entertainment offering, comprising: receiving, by a controller, asensor signal from a sensor device, the sensor signal including at leastone image of a current interior situation of the motor vehicle;generating, by the controller, a superposition signal instructingsuperposition of the at least one image of the current interiorsituation on the specified output content; and transmitting thesuperposition signal from the controller to a display element of thedisplay device, thereby causing the display device to superimpose the atleast one image of the current interior situation on the specifiedoutput content.
 12. The method as claimed in claim 11, wherein thesuperposition signal instructs the display device to superimpose the atleast one image of the current interior situation in only a partialregion of a visual field of a user of the display device.
 13. The methodas claimed in claim 12, wherein the superposition signal instructs thedisplay device to superimpose the at least one image of the currentinterior situation in one of a primary visual field and a central visualfield.
 14. The method as claimed in claim 12, wherein the sensor signalprovides at least one camera image of an interior camera of the sensordevice.
 15. The method as claimed in claim 14, wherein the controllerreceives the sensor signal from the sensor device of the motor vehicle.16. The method as claimed in claim 14, wherein the specified outputcontent defines a 360-degree virtual reality.
 17. The method as claimedin claim 14, further comprising determining, by the controller, that anoccupant is speaking, as a prerequisite for said transmitting of thesuperposition signal.
 18. The method as claimed in claim 17, wherein theoccupant determined to be speaking is not the user of the displaydevice.
 19. The method as claimed in claim 18, wherein said determiningis in dependence on a volume of a voice of the occupant.
 20. The methodas claimed in claim 17, further comprising determining, by thecontroller, whether the motor vehicle is approaching a destination, andwherein at least one of said generating and said transmitting of thesuperposition signal is performed after determining that the motorvehicle approaches the destination.
 21. The method as claimed in claim11, wherein the superposition signal instructs the display device tosuperimpose the at least one image of the current interior situation inonly one of a primary visual field and a central visual field.
 22. Themethod as claimed in claim 11, further comprising determining, by thecontroller, that an occupant is speaking, as a prerequisite for saidtransmitting of the superposition signal.
 23. The method as claimed inclaim 22, wherein the occupant determined to be speaking is not the userof the display device.
 24. The method as claimed in claim 23, whereinsaid determining is in dependence on a volume of a voice of theoccupant.
 25. A controller in a motor vehicle where a sensor deviceoutputs a sensor signal and a display device displays specified outputcontent of an entertainment offering, said controller comprising: amemory; and a processor, connected to the memory, configured to receivethe sensor signal from the sensor device, the sensor signal including atleast one image of a current interior situation of the motor vehicle,generate a superposition signal instructing superposition of the atleast one image of the current interior situation on the specifiedoutput content, and transmit the superposition signal from thecontroller to the display device, causing the display device tosuperimpose the at least one image of the current interior situation onthe specified output content.
 26. The controller as claimed in claim 25,wherein the method further comprises determining that an occupant isspeaking, before the superposition signal is transmitted.
 27. A motorvehicle, where a display device displays specified output content of anentertainment offering, comprising: a sensor device configured to outputa sensor signal including at least one image of a current interiorsituation of the motor vehicle; and a controller configured to receivethe sensor signal from the sensor device, generate a superpositionsignal instructing superposition of the at least one image of thecurrent interior situation on the specified output content, and transmitthe superposition signal from the controller to the display device,causing the display device to superimpose the at least one image of thecurrent interior situation on the specified output content.
 28. Themotor vehicle as claimed in claim 27, wherein the controller is furtherconfigured to determine whether the motor vehicle is approaching adestination, and at least one of generate and transmit the superpositionsignal after determining that the motor vehicle approaches thedestination.
 29. A head-mounted display device, operated in a motorvehicle having a sensor device configured to output a sensor signalincluding at least one image of a current interior situation of themotor vehicle; said head-mounted display device comprising: a displayelement; and a controller configured to. display specified outputcontent of an entertainment offering on the display element, receive thesensor signal from the sensor device, generate a superposition signalinstructing superposition of the at least one image of the currentinterior situation on the specified output content, and transmit thesuperposition signal from the controller to the display device, causingthe display device to superimpose the at least one image of the currentinterior situation on the specified output content.
 30. The head-mounteddisplay device as claimed in claim 29, wherein the superposition signalinstructs the display device to superimpose the at least one image ofthe current interior situation only in one of a primary visual field anda central visual field, forming less than all of a total visual field ofa user of the display device.